Add MMQ v2 and generic model adapters - #21

Purpose

Combine the GGUF plugin work needed for current Gemma4 and Qwen3.6 GGUF serving experiments:

1. add the MMQ v2 CUDA path for IQ4_XS batched GGUF inference,
2. resolve GGUF config sources through a generic local/remote fallback chain,
3. add Gemma4 and Qwen3.5/Qwen3.6 GGUF tensor mapping adapters,
4. wire GGUF MTP config and weight loading into the plugin path.

This is a fork-side integration PR only. It is not opened against upstream yet.

1. MMQ v2 Kernel Path

Adds a vLLM-native MMQ v2 CUDA path under the plugin _C_gguf namespace.

Main changes:

• add mma_v2.cuh / mmq_v2.cuh device helpers,
• register stable torch ops for:
  • ggml_mul_mat_a8_q4_0_mmq_v2 as a reference/test path,
  • ggml_mul_mat_a8_iq4_xs_mmq_v2 for dense linear IQ4_XS,
  • ggml_moe_a8_iq4_xs_mmq_v2 for MoE IQ4_XS legs,
• route IQ4_XS batched dense linear through MMQ v2 while keeping batch-size-1 MMVQ unchanged,
• route IQ4_XS MoE legs through MMQ v2 only for batched MoE input,
• keep existing standard/K-quant MMQ and MMVQ paths unchanged.

Q4_0 v2 is included to validate the modern MMA path against an already-supported quant type. It does not replace the existing Q4_0 production dispatch in this PR.

2. GGUF Config Resolution

Adds a generic config-source resolver for GGUF models.

Fallback order:

1. local GGUF directory,
2. local parent directory for nested GGUF files such as MTP/*.gguf,
3. remote GGUF repository,
4. base_model from GGUF metadata/model card when it points to a valid HF repo,
5. existing vLLM fallback behavior.

The resolver validates remote base model IDs and disables trust_remote_code when configuration is redirected from a GGUF-derived source. This keeps config discovery generic without trusting code from a repo selected by GGUF metadata.

3. Gemma4 and Qwen GGUF Adapters

Adds plugin-side tensor mapping and weight-loading adapters for current GGUF layouts.

Covered paths:

• Gemma4 text/MM tensor name mapping,
• Gemma4 patch embedding and qweight type loading behavior,
• Qwen3.5/Qwen3.6 tuple shard loading for fused projections,
• Qwen3.5/Qwen3.6 MoE expert sideload mappings,
• GGUF metadata patching before adapter selection.

The model-specific logic stays in plugin adapters instead of patching vLLM core model files.

4. GGUF MTP Support

Adds GGUF MTP config and weight loading support for the current Qwen3.6 and Gemma4 GGUF layouts.

Covered paths:

• pass GGUF config source resolution into the speculator override flow,
• patch Qwen MTP metadata from GGUF keys such as nextn_predict_layers,
• patch Gemma4 MTP metadata from GGUF keys such as embedding_length, embedding_length_out, attention head counts, and layer metadata,
• map appended Qwen MTP blocks into mtp.layers.*,
• map Gemma4 assistant MTP tensors into Gemma4MTPModel,
• initialize the GGUF-loaded model with the resolved model_config.

Test Plan

PYTHONPATH=/nvme512g/worktrees/vllm-gguf-mmq-v2-stack-44905-20260610:/tmp/vllm-gguf-plugin-pr89 \
  /nvme512g/venvs/vllm-pr86-cuda/bin/python -m pytest tests/test_plugin.py -q

PYTHONPATH=/nvme512g/worktrees/vllm-gguf-mmq-v2-stack-44905-20260610:/tmp/vllm-gguf-plugin-pr89 \
  /nvme512g/venvs/vllm-pr86-cuda/bin/python -m ruff check vllm_gguf_plugin tests/test_plugin.py

PYTHONPATH=/nvme512g/worktrees/vllm-gguf-mmq-v2-stack-44905-20260610:/tmp/vllm-gguf-plugin-pr89 \
  /nvme512g/venvs/vllm-pr86-cuda/bin/python -m ruff format --check vllm_gguf_plugin tests/test_plugin.py

PYTHONPATH=/nvme512g/worktrees/vllm-gguf-mmq-v2-stack-44905-20260610:/tmp/vllm-gguf-plugin-pr89 \
  /nvme512g/venvs/vllm-pr86-cuda/bin/python -m compileall vllm_gguf_plugin

git diff --check

Test Result

• tests/test_plugin.py: 26 passed
• ruff check: passed
• ruff format --check: passed
• compileall: passed
• git diff --check: passed
• Gemma4 MTP GGUF parser smoke:
  • architectures = ['Gemma4MTPModel']
  • model_type = gemma4_assistant
  • num_hidden_layers = 4
  • mtp_num_hidden_layers = 4
  • hidden_size = 1024
  • backbone_hidden_size = 2816
• Qwen3.6 MTP GGUF parser smoke printed the expected patched metadata before the local command was stopped during remote config lookup:
  • model_type = qwen3_5_moe
  • text.num_hidden_layers = 40
  • mtp_num_hidden_layers = 1
  • num_nextn_predict_layers = 1
  • hidden_size = 2048

Local kernel validation from the integration worktree:

• Q4_0 linear MMQ v2 correctness passed; large-batch path measured about 5-7x faster than the old b2899 MMQ path.
• IQ4_XS dense linear MMQ v2 correctness passed; measured about 1.15-3.2x faster than dequant fallback depending on shape.
• IQ4_XS MoE MMQ v2 correctness passed; measured about 2.1-10x faster than the existing MoE vector path depending on model shape.
• MoE top_k=1, all-expert-0 gate matched linear v2 with max diff 0.

Notes

• ROCm is not verified in this local run.
• End-to-end serving validation is still separate from this fork-side integration PR.
• This PR intentionally keeps existing standard/K-quant MMQ dispatch unchanged.

AI assistance: Codex and Claude.